Usage of GBOD Functions¶
In [118]:
%matplotlib notebook
%load_ext autoreload
%autoreload
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import OutlierDetection as od
1. Read the dataset df from original combine harvester with machineId 903¶
In [151]:
# set the dataset path
inputPath = "D:/project/agata/data/olddata/"
inputFile = inputPath+"58900903_telematics_agata_v3__final.csv"
# read the dataset and drop the missing value
df_full = pd.read_csv(inputFile)
df_full.head()
#df_full.columns # List all the columns
Out[151]:
In [120]:
df = df_full[["ertrag","durchsatz"]].dropna()
# show the head of the input dataset
df.head()
Out[120]:
In [121]:
df_full.columns
Out[121]:
2. Usage Examples¶
2.1 Set the parameter and initialize a GBOD Object¶
In [122]:
n_partition = 20 # number of partition in each attribute
outlier_percent = 0.5 # percent of outliers
myGBOD = od.GBOD(df, n_partition=n_partition,outlier_percent=outlier_percent) # create the GBOD object
2.2 use the functions¶
In [123]:
# run the function `run_GBOD()` to get the result.
result, grid_info = myGBOD.run_GBOD()
In [124]:
# result is the dataframe with all the information of the points.
result.head()
Out[124]:
In [125]:
# `grid_info` is a dataframe with all grid information
grid_info.head()
Out[125]:
In [126]:
# You can also set the different percent of outliers and write down the result as txt file
result, grid_info = myGBOD.run_GBOD(outlier_percent=0.25, writeCSV=True)
2.3 Use the center_bias function¶
Center Bias means the difference between point center and grid center
In [127]:
result_cb, grid_info_cb = myGBOD.run_GBOD_center_bias(outlier_grid_frac=0.5)
result_cb.head()
Out[127]:
In [128]:
grid_info_cb.head()
Out[128]:
2.4 Plot¶
In [129]:
# Plot the histogram about number of points in each grid
myGBOD.plotGridHist(bins = 50)
In [130]:
myGBOD.np_in_grid.max()
Out[130]:
In [131]:
# Plot the heatmap
myGBOD.plotGridHeatMap(fontsize=12)
In [132]:
# plot the points with different outlier_percent
myGBOD.plotOutliers()
myGBOD.plotOutliers(grid=False)
In [133]:
# plot the points in original dataset
myGBOD.plotOutliers_origin(outlier_percent=0.5, grid=False)
In [134]:
#plot the normed dataset with point center for each grid
myGBOD.plot_df_norm()
In [135]:
# plot the outliers after center bias method with different outlier_grid_frac
myGBOD.plotOutliers_CenterBias()
3. Other Examples¶
3.1 Sample a part of records from a dataset¶
Sample a given number of records
In [136]:
# Sample 4000 records from df
mySample = od.sample_df(df, n=4000, random_state=1)
mySample.head()
Out[136]:
In [137]:
# Sample 1% records from df
mySample = od.sample_df(df, frac=0.01, random_state=1)
mySample.head()
Out[137]:
3.2 Sample 7000 records from df and run gbod¶
In [138]:
# sample 7000 records from df
df7000=od.sample_df(df, n=7000, random_state=1)
# initialize the GBOD Object with 15 partition on each attribute.
myGBOD2 = od.GBOD(df7000, n_partition=20, outlier_percent=1)
# get result
result,grid_info = myGBOD2.run_GBOD()
result.head()
Out[138]:
In [139]:
# get the third and fourth column for the normalized dataset
myGBOD2.res.loc[:, ["ertrag_norm", "durchsatz_norm"]].head()
Out[139]:
In [140]:
myGBOD2.np_in_grid.nlargest()
Out[140]:
In [141]:
# plot the result
myGBOD2.plotOutliers()
# plot the result with grid
myGBOD2.plotOutliers(grid=False)
In [142]:
myGBOD2.plotGridHeatMap(fontsize=14)
3.3 KDTree-KNN Example¶
In [143]:
mySample=od.sample_df(df, n=10000, random_state=1)
In [144]:
myKDKNN = od.KDKNN(mySample, k=5)
In [145]:
myKDKNN.run_KDKNN(outlier_percent=1).head()
Out[145]:
In [146]:
myKDKNN.plot_outliers(grid=True, outlier_percent=1)
3.4 GBOD vergleich mit KDKNN¶
Create GBOD3¶
In [147]:
myGBOD3 = od.GBOD(mySample, n_partition=20, outlier_percent=5)
Vergleich mit KDKNN mit function compare_KNKNN(outlier_percent)¶
In [148]:
myGBOD3.compare_KDKNN()
Out[148]:
In [149]:
myGBOD3.compare_KDKNN(outlier_percent=2)
Out[149]:
3.5 measure the time¶
GBOD¶
In [161]:
%timeit myGBOD.findPartition()
KDKNN¶
In [158]:
%timeit od.get_KDKNN_score(df)